Node timing #747

paigerube14 · 2025-01-14T15:25:24Z

This is adding in the ability to add AffectedNode timing to a list that we can track in the telemetry and other output.

This tracks the amount of time that is taken for the cloud provider to stop/start the node, and the amount of time after the node from the cloud side is stopped/started what is the time the node is in not ready/ready state.

Needs to go in after krkn-chaos/krkn-lib#143

Any suggestions on how to make this not touch as many files?

New telemetry section will look like this for a stop/start scenario

          "affected_nodes": [
                    {
                        "node_name": "ip-*.us-east-2.compute.internal",
                        "not_ready_time": 0.0,
                        "ready_time": 24.439035892486572,
                        "unknown_time": 0.15461111068725586,
                        "stopped_time": 136.35135626792908,
                        "running_time": 15.249454021453857,
                        "terminating_time": 0.0
                    },
                    {
                        "node_name": "ip-*.us-east-2.compute.internal",
                        "not_ready_time": 0.0,
                        "ready_time": 23.62007188796997,
                        "unknown_time": 0.15206694602966309,
                        "stopped_time": 166.52288508415222,
                        "running_time": 15.330392122268677,
                        "terminating_time": 0.0
                    }
                ],

chaitanyaenr

Tested different combinations and the reporting is working as expected. Working node scenario configs:

one action - node_stop_start_scenario targeting one instance count
one action - node_stop_start_scenario targeting multiple instance count

Scenario with multiple actions is failing to report the metrics due to a bug in the node-scenarios code base outside this PR: #749

chaitanyaenr · 2025-01-18T05:27:59Z

krkn/scenario_plugins/node_actions/node_actions_scenario_plugin.py

@@ -247,6 +253,8 @@ def run_node(self, single_node, node_scenario_object, action, node_scenario):
                    "There is no node action that matches %s, skipping scenario"
                    % action
                )
+        logging.info('last line run node')


Leftover logging from the test runs, need to remove it.

paigerube14 added 7 commits January 10, 2025 11:38

start of node timing

67a22d2

adidng node scenario updates

3340a6f

adding node action changes for affected nodes

d3483ca

adding node actions with timign

0e4e4a9

adding cluster shut down with node timing

7ffdfc7

adding few more small change

6eedca2

adding back in cluster-events

cef3e03

paigerube14 requested review from chaitanyaenr and tsebastiani January 14, 2025 15:25

paigerube14 added 3 commits January 14, 2025 12:27

updating effected to affected in aws

8ef90ab

using event backup to collect events or not

2a73145

adding extend

0e5ea38

chaitanyaenr reviewed Jan 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Node timing #747

Node timing #747

paigerube14 commented Jan 14, 2025 •

edited

Loading

chaitanyaenr left a comment

chaitanyaenr Jan 18, 2025

Node timing #747

Are you sure you want to change the base?

Node timing #747

Conversation

paigerube14 commented Jan 14, 2025 • edited Loading

chaitanyaenr left a comment

Choose a reason for hiding this comment

chaitanyaenr Jan 18, 2025

Choose a reason for hiding this comment

paigerube14 commented Jan 14, 2025 •

edited

Loading